Skip to main content

Incident Report: ETH/USD Out of Deviation on Blast Sepolia Testnet

Date: 2024-04-02
Time: 10:19 (UTC+05:30)
Duration: 14 hours

Description​

There were consecutive ETH/USD out of deviation alerts on Blast Sepolia Testnet (stage) in a regular interval because of funding.

Root Cause​

The feed is expiring in a day. It's underpriced and the wallet ran out of gas 4 days before its expiration date.

Impact​

We got a critical deviation alert which could potential commotion that actually is not to be considered that of harm. Alert behavior is also not appropriate to update once funded. This is not an issue on testnet though but worth changing the behaviour on the real thing.

Timeline​

  • 10:19 - First noticed the issue.
  • 17:15 - Initial diagnosis.
  • 12:19 (DAY 2) - Started the fix.
  • 01:30 (DAY 2) - Issue resolved.

Lessons Learned​

It might be better if it's re-priced before the next buy. Alerts behavior should be appropriate to see through all updates.

Actions Taken​

Burak checked the scripts and funded the wallet.

Slack escalation link.

Incident Reviewer(s)​

  • Andrew
  • Warren
  • Ali
  • Burak
Last updated on by Warren Anderson